1. Field of the Invention
The present invention relates to using computing devices to directly generate a virtual view of a scene. More particularly, the present invention relates to a direct synthesis approach using layers in order to directly generate a virtual view from a pair of images.
2. Description of Background Art
The field of image-based rendering (IBR) covers using a set of input images of a scene from particular viewpoints to generate a new (synthetic) image of the scene from a new viewpoint. IBR techniques can be classified based on how a scene's geometry is used and represented. At one end of the spectrum, there are techniques that require little or no geometric information about the scene, such as the Light Field and Lumigraph methods. These techniques require special capturing equipment and use a very large input image database, even for very small objects. Thus, they are called dense IBR methods. Dense IBR methods have the disadvantage of requiring a fairly large number of input images, high equipment complexity, and a large amount of image storage.
At the other end of the spectrum, there are techniques that use relatively few input images, but require more geometric information about the scene. These “sparse IBR” methods use a set of input images and their associated depth maps to render a synthetic image. When depth information is available for every point in an input image, the image can be rendered from any nearby point of view by projecting the pixels of the input image into the scene and then re-projecting these pixels onto the synthetic image. Fewer input views means simpler equipment setups and smaller storage requirements. However, the quality of the synthetic view depends heavily on the accuracy of the geometric information.
One sparse IBR approach is to compute a depth representation using a stereo algorithm. Stereo techniques must solve the Stereo Correspondence problem, where each pixel in a first input image must be matched (i.e., recognized as the same object) to a pixel in a second input image. Traditional stereo algorithms cannot fully solve the Stereo Correspondence problem for objects of uniform color and texture because of matching ambiguity. As a result, stereo algorithms fail to recover the accurate depth of objects in this situation. Moreover, stereo techniques applied to IBR attempt to compute the depth at every pixel of an input image, regardless of whether the pixel is used in the final synthetic view. Also, computing high-quality synthetic images using stereo-based approaches can be computationally expensive.
An alternative to the traditional stereo-based approach is to perform calculations from the point of view of the synthetic view. This is called direct-view synthesis, and depth is computed at only the locations that are relevant to the rendering of the synthetic image. An advantage of this approach is that accurate geometry is not needed for surfaces of uniform color and texture. That is, methods using the direct-view synthesis approach can afford to make mistakes in the depth calculation at those locations where geometry is unimportant. Thus, it is irrelevant that the stereo correspondence problem becomes difficult in those situations.
One direct-view synthesis method is the Range-Space Approach, which uses a voxel representation of the scene. This method casts a viewing ray from the virtual viewpoint for every pixel in the synthetic image, cutting through the voxel representation of the scene. The problem consists in finding the voxel that corresponds to a physical surface. This is indicated when the neighborhood around a voxel is colored in a similar fashion by all input views. The procedure requires a volumetric matching template to compute scores along the viewing ray. The method has the advantage of producing directly-synthesized virtual views. But, like other voxel-based methods, it is computationally expensive and unlikely to compute high-quality synthetic views in real-time.
What is needed is a direct-view synthesis method that avoids the problems of the Range-Space Approach.